Exponential backoff with jitter should work fine for most cases, but assuming no jitter in the case of 3 retries that's a call at 0, 150ms, 600ms, and 1200ms (am I correct here?)

It would be nice to see if this covers every case, or if there would still be a super small chance that this error might get hit (even with the exponential backoff).

Timestamp failed to pass the watermark after retrying the configured 3 times. We cannot guarantee the integrity of the response since it could be a replay attack.

If this error got hit with the exponential backoff in place, what would that mean for the FE app? Is the subnet down or behind? Did we just get super unlucky? Does the same error message make sense now that the exponential backoff is in place (i.e. does this error have a different meaning)?

Ideally, this error would never happen by accident, or even if the FE client user has a poor internet connection. Then if they receive the error it would actually suggest something is not healthy with the nodes/subnet they've communicated with.

Putting a higher threshold in by using exponential backoff and hoping that this is a rare case is just going to confuse developers, and especially confuse end users in the even this error is hit and a toast component pops up with "Timestamp failed to pass the watermark after retrying the configured 3 times. We cannot guarantee the integrity of the response since it could be a replay attack" (in fact, this sort of sounds like security issue).

LGTM